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Abstract 

This  purpose  of  this  study  is  to  examine  the  evolution  of  the  Acquisition  Review 
Journal  through  its  first  eleven  years  in  publication.  Researchers  will  assess  the  Defense 
Acquisition  community  through  a  review  of  ARJ  articles.  It  considers  what  areas 
academics  and  practitioners  have  explored  and  how  they  have  done  so.  This  review 
documents  such  characteristics  as  areas  of  study,  methods  of  study,  and  contributors. 
Trends  are  identified  and  conclusions  drawn  as  to  the  contribution  of  ARJ  to  the  Defense 
Acquisition  community  of  practice. 
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AN  ELEVEN  YEAR  RETROSPECTIVE  OF  THE  ACQUISITION  REVIEW 

JOURNAL 


I.  Introduction 


Background 

For  the  last  several  years,  great  emphasis  has  been  placed  on  refonning 
Department  of  Defense  (DoD)  Acquisition  practices.  Several  factors,  including  ever- 
changing  threats  and  cuts  to  personnel  and  budgets,  contribute  to  this  increased  emphasis. 

In  an  effort  to  document  and  further  acquisition  refonn  throughout  DoD,  the 
Acquisition  Review  Journal  (formerly  Acquisition  Review  Quarterly)  was  established. 

In  its  inaugural  issue,  the  Deputy  Under  Secretary  of  Defense  for  Acquisition  Reform 
introduced  the  journal  as  one  way  to  raise  awareness  of  the  Acquisition  career  field, 
arguing  that  greater  awareness  was  a  necessary  step  in  furthering  reform  efforts  (Preston 
1994).  Since  then,  ARJ  has  become  the  flagship  professional  publication  of  the  Defense 
acquisition  community. 

Along  with  creating  awareness  in  a  particular  field  of  study,  journals  also  serve  as 
a  forum  for  intellectual  exchange  within  a  community  of  practice,  and  assessing  the 
content  of  journals  is  one  way  to  evaluate  a  discipline’s  intellectual  health  (Das  and 
Handheld,  1997).  The  Acquisition  Review  Journal  is  a  forum  for  intellectual  exchange 
created  specifically  for  DoD  acquisition  refonn  and  therefore  a  study  of  its  content  could 
offer  an  appropriate  assessment  of  DoD  acquisition  reform. 
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The  purpose  of  this  study  is  to  examine  the  evolution  of  the  Acquisition  Review 
Journal  through  its  first  eleven  years  of  existence.  The  study  assesses  the  state  of  the 
Defense  acquisition  community  of  practice  through  a  review  of  ARJ  articles.  It  seeks  to 
understand  what  areas  academics  and  practitioners  have  explored  and  how  they  have 
done  so.  This  review  will  document  such  characteristics  as  areas  of  study,  methods  of 
study,  and  contributors.  Trends  will  be  identified  and  conclusions  drawn  as  to  the 
contribution  of  ARJ  to  the  Defense  Acquisition  community  of  practice. 

History  of  Acquisition  Review  Journal 

Researchers  conducted  a  brief  history  of  editors  in  the  ARJ.  Specifically,  they 
looked  at  the  editors  over  the  initial  eleven  years  and  reviewed  the  changes  in  guidance 
provided  for  perspective  authors.  In  eleven  years  of  publishing  the  ARJ  there  have  been 
four  editors.  Table  1  shows  the  journal  editors  and  their  term  of  editorship. 


Table  1.  Journal  Editorship 


Time  Period 

Editor 

Winter  1994-Spring  1995 

Robert  W.  Ball 

Summer  1995-Fall  1997 

James  Kurt  Wittmeyer 

Winter  1997-  Summer  2000 

Deborah  L.  Gonzalez 

Fall  2000-Present* 

Norene  L.  Taylor** 

*For  the  purpose  of  this  study  “present”  is  considered  the  end  of  2004 
**  Norene  L.  Taylor  is  listed  under  the  following  surnames:  Blanch,  Fagan-Blanch,  and  Taylor. 
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Researchers  also  conducted  a  review  of  the  various  documents  provided  as 
guidance  for  authors  in  order  to  trace  a  history  of  submission  requirements  for 
contributions. 

In  the  inaugural  issue,  Winter  1994,  the  Guidelines  for  Authors  states  that  the 

ARJ  is  looking  for  articles  that  “represent  scholarly  examination,  disciplined  research  and 

supported  empirical  experience  in  the  fields  of  defense  systems  management  and 

acquisitions  management.  Defense  acquisition  is  the  primary  focus,  but  papers  covering 

other  fields  of  management  will  be  considered.”  In  Summer  1994,  editors  published  their 

Editorial  Mission.  The  mission  asserts  the  ARJ’s  intentions  and  purpose  of  publishing 

articles  and  the  type  of  infonnation  the  ARJ  will  disseminate.  The  goal  of  the  ARJ  is 

stated  in  the  excerpt  below,  taken  from  the  Editorial  Mission: 

The  primary  goal  of  the  Acquisition  Review  Quarterly  (ARQ)  is  to  provide 
practicing  acquisition  professionals  with  relevant  management  tools  and 
information  based  on  recent  advances  in  policy,  management  theory,  and 
research. .  .and  is  intended  to  serve  as  a  mechanism  for  fostering  and 
disseminating  scholarly  research  on  acquisition  issues,  for  exchanging  opinions, 
for  communicating  policy  decisions,  and  for  maintaining  a  high  level  of 
awareness  regarding  acquisition  management  philosophies.  The  ARQ  provides 
insight  to  the  acquisition  professional  and  others  in  the  Department  of  Defense 
(DoD),  Congress,  industry  and  academe  who  have  significant  interest  in  how  the 
DoD  conducts  its  acquisition  mission.  (ARQ,  1994:3) 

In  the  Summer  1995  issue,  the  ARJ  simplified  their  requests  for  contributors  by 
welcoming  “anyone  interested  in  the  defense  acquisition  process.”  It  also  asks  for 
articles  under  specific  heading  types  such  as  research,  policy,  and  tutorials.  In  previous 
issues,  these  headings  are  not  mentioned. 
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The  Winter  1997  issue  marked  the  first  time  the  ARJ  requested  articles  with  a 
specific  topic,  as  opposed  to  the  previous  suggestion  of  topic  areas.  This  request  was  a 
call  for  manuscripts  on  Radical  Change  in  Defense  Acquisitions. 

The  Spring  1997  issue  showed  a  turning  point  in  the  ARJ’s  guidelines  for  authors. 
The  ARJ  still  stated  that  article  submissions  are  welcome  from  anyone  interested  in 
defense  acquisitions,  but  a  list  of  examples  follows.  This  list  includes: 
“conceptualization,  initiation,  design,  development,  test,  contracting,  production, 
deployment,  logistic  support,  modification,  and  disposal  of  weapons  and  other  systems, 
supplies,  or  other  services  to  satisfy  Defense  Department  needs,  or  intended  for  use  in 
support  for  use  in  support  of  military  missions.”  The  guidelines  for  this  issue  also 
include  specific  instructions  on  manuscript  sections  for  each  type  of  article  (Research, 
Opinion,  and  Tutorial).  This  issue  has  more  style  guidance  than  any  previous  issue. 

The  most  recent  significant  change  in  editorial  guidelines  occurred  in  the  Winter 
2003  issue;  the  ARJ  Guidelines  for  Authors  suggest  that  articles  be  co-authored  to  add 
depth,  and  that  a  mentor  who  has  published  before  or  has  expertise  in  the  subject  area  be 
selected  to  assist  with  the  submission.  This  suggestion  is  reiterated  in  succeeding  issues. 

Research  Problem 

The  ARJ  has  been  in  publication  for  eleven  years  as  a  way  to  exchange  ideas 
within  the  defense  acquisition  community  and  to  document  the  Acquisition  Reform 
movement  in  the  DoD.  While  each  issue  of  the  ARJ  explores  individual  reform  efforts 
and  ideas,  researchers  could  find  nothing  written  to  examine  the  journal  as  a  whole.  One 
article  (Rogers  and  Birmingham,  2004)  examined  the  history  of  acquisition  reform 
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efforts,  but  this  article  focused  mainly  on  what  the  authors  consider  “landmark 
documents.”  The  authors  identified  eight  such  documents  they  used  to  frame  the  reform 
process  and  examined  reform  efforts  across  several  publications,  not  specifically  the 
ARJ.  This  study  seeks  to  systematically  analyze  the  contributions  of  the  ARJ  in  order  to 
understand  acquisition  practice  in  the  DoD. 

Investigative  Questions 

Several  research  questions,  based  on  the  content  analysis  methodologies 
employed  by  Carter  and  Ellram  (2003)  and  Das  and  Handfield  (1997),  have  been 
developed  to  guide  the  study. 

1.  What  subjects  have  been  addressed  in  the  ARJ?  What  do  these  topics  suggest 
about  acquisition  reform  in  the  DoD? 

2.  How  have  these  subjects  been  explored? 

a.  What  research  methodologies  have  been  used  to  explore  the  subjects, 
and  to  collect  data? 

b.  What  analytical  techniques  have  been  used  on  the  data? 

c.  What  do  these  findings  suggest  about  the  study  of  acquisition  reform  in 
the  ARJ? 

3.  What  authors  have  contributed  articles  to  ARJ,  and  with  what  institutions  are 
they  affiliated?  What  does  this  suggest  about  the  study  of  acquisition  reform? 

Methodology 
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Similar  studies  have  contributed  to  the  bodies  of  knowledge  in  related  fields 
(Carter  and  Ellram  2003;  Das  and  Handheld  1997).  This  study  will  loosely  replicate  the 
methodology  employed  by  Carter  and  Ellram  (2003)  in  their  retrospective  of  the  Journal 
of  Supply  Chain  Management,  and  that  used  by  Das  and  Handheld  (1997)  in  their  review 
of  the  Journal  of  Operations  Management.  Each  article  published  in  the  ARJ  will  be 
carefully  examined  and  classified  based  upon  predetermined  characteristics  such  as 
subject  matter  and  methodology.  Trends  will  be  identified  across  characteristics  as 
appropriate  and  conclusions  will  be  drawn  as  to  ARJ’s  contribution  to  the  Defense 
Acquisition  community  of  practice. 

Proposed  Study  Contributions 

The  results  may  provide  Acquisition  professionals  with  a  better  understanding  of 
how  the  journal  has  explored  reform.  The  results  will  reveal  the  subjects  that  have  been 
addressed  in  the  ARJ,  how  the  subjects  were  explored,  how  data  was  collected  and 
analyzed,  and  who  contributes  to  the  ARJ.  The  results  of  this  study  might  also  be  helpful 
in  infonning  future  authors  how  they  can  improve  their  research  and  publications,  thus 
contributing  to  the  advancement  of  knowledge  about  defense  acquisitions.  Finally,  the 
results  might  provide  some  indications  about  where  the  Journal  should  go  next. 


Overview 
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This  chapter  introduced  the  specific  area  of  study.  It  provided  some  background 
on  the  Acquisition  Review  Journal.  Following  the  background,  there  was  a  brief 
discussion  of  the  problems  to  be  addressed  in  this  study  and  some  investigative  questions. 
A  proposed  methodology  was  presented  and  study  contributions  were  considered.  The 
remainder  of  this  thesis  is  structured  as  follows:  Chapter  Two  will  present  the  method, 
Chapter  Three  will  present  the  analysis,  and  Chapter  Four  will  present  conclusions. 


II.  Methodology 
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Chapter  two  describes  the  methodology  employed  in  this  research  project. 


Research  Design 

This  study  was  a  systematic  examination  of  the  content  of  the  ARJ  (over  the  first 
eleven  years  of  publication)  using  Content  Analysis  to  identify  patterns  or  themes.  Open 
Coding  was  used  to  categorize  the  data  and  examine  the  data  for  properties  and  specific 
attributes  (Leedy  and  Onnrod,  2001).  There  were  193  articles  used  in  the  study.  These 
were  journal  articles  found  at  www.dau.mil/pubs/arqtoc.asp.  It  is  important  to  note  that 
Editorial  Introductions  and  Special  Edition  Introductions  were  not  included  in  the  data. 

Article  Subject  Matter 

Two  researchers  initially  went  through  the  text  of  all  the  articles  in  the  journal. 
Each  article  was  carefully  scrutinized  to  identify  subjects  under  which  the  article  could 
be  categorized.  Where  applicable,  articles  that  had  more  than  one  conceivable  subject 
category  were  coded  with  all  possible  subject  categories.  This  review  identified  126 
possible  subjects.  Next,  the  researchers  independently  grouped  the  subjects  based  on 
common  themes.  This  initial  trial  yielded  39  categories.  The  researchers  then 
accomplished  a  second  independent  grouping  in  order  to  identify  broader  categories. 

This  trial  yielded  a  70%  agreement  rate  between  coders.  Differences  in  categorization 
were  resolved  through  discussions  in  order  to  reach  a  consensus  on  the  final  15 
categories.  The  final  subject  categories  were  defined  (see  Table  2). 


Table  2.  Definitions  of  Subject  Categories 


Subject  Categories 

Definition 

Management  and  Organizational  Behavior 

Concerns  leadership/management  theory, 
workforce  development,  and  recruiting  and 
retention 

Organizational  Issues 

Concerns  organizational  restructuring  and 
organizational  strategy 

Policy  and  Regulation 

Encompasses  acquisition  regulation  and 
public  policy  issues 

Performance  and  Measurement 

This  category  contains  articles  related  to 
performance  measurement  and  metrics 

Buyer/Seller  Relationships 

Concerns  supplier  development,  supply 
chain  issues,  and  partnering 

Research  and  Development 

Entails  subjects  related  to  new  research 
endeavors  not  directly  related  to 
interoperability  or  program  fielding 

Acquisition  Strategy 

Contains  articles  with  big  picture  views  in 
acquisition  strategy  such  as  outsourcing, 
privatization,  and  cooperative  acquisitions 

Interoperability 

Includes  technology  integration,  information 
technology,  and  system  architecture  and 
design 

Risk  Management 

Contains  articles  related  to  risk  management 
issues 

Cost  and  Schedule 

Concerns  issues  of  cost  and  schedule  such  as 
growth  and  variance 

Analysis  and  Decision  Making 

Articles  related  to  analysis  models  in 
management  decisions,  cost  analysis,  and 
budgeting 

Industry  Issues 

Includes  articles  addressing  the  defense 
industry  and  articles  on  commercial  issues 
with  a  industry  base  focus 

Reform  Initiatives 

Contains  articles  addressing  acquisition 
reform  such  as  best  practices,  efficiency 
initiatives,  quality,  and  articles  with 
commercial  focus  on  reform 

Program  Fielding  and  Implementation 

Articles  concerned  with  program  fielding 
and  implementation  issues  such  as  field 
testing  and  battle  labs 

Contract  Management 

Articles  related  to  contract  management, 
contingency  contracting,  source  selection 
decisions,  acquisition  planning,  and  contract 
incentives 

Once  the  categories  were  defined,  the  researchers  recoded  each  article  into  one  of 
these  final  categories.  Again,  differences  among  researchers  were  resolved  through 
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discussion.  The  data  was  compiled  and  tables  were  established  in  order  to  detennine  the 
relative  frequency  with  which  publications  appeared  over  time.  In  order  to  gain  a  better 
insight  of  trends,  researchers  divided  the  eleven  years  into  three  periods.  Period  One 
covered  Years  1  through  4  and  contained  65  articles,  Period  Two  contained  Years  5 
through  7  with  65  articles,  and  Period  Three  covered  Years  8  through  1 1  with  63  articles. 
The  data  was  then  analyzed  for  trends  across  subject  and  time  period. 

Article  Methodology 

Articles  were  classified  according  to  the  methodology  employed.  Method 
categories  used  were  based  on  those  employed  by  Carter  and  Ellram  (2003),  who 
followed  Mentzer  and  Kahn’s  (1995)  typologies.  Categories  used  include  Type  of 
Research  Perfonned,  Type  of  Design  Employed,  and  Data  Analysis  Employed  (Carter 
and  Ellram,  2003). 

Type  of  Research  Performed 

The  articles  were  coded  using  Mentzer  and  Kahn’s  (1995)  typology.  This 
typology  is  based  on  five  categories  including:  Normative  Literature,  Literature  Reviews, 
Exploratory  Studies,  Methodology  Reviews,  and  Hypothesis  Testing.  The  coders 
reviewed  the  articles  and  detennined  which  category  was  most  appropriate  for  each 
article.  Researchers  then  created  a  table  displaying  the  frequency  of  articles  per  category. 
The  data  was  compiled  and  tables  were  established  in  order  to  determine  the  relative 
frequency  with  which  various  methodologies  were  used  over  time.  To  gain  a  better 
insight  of  trends,  researchers  divided  the  eleven  years  into  three  periods.  The  data  was 
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then  analyzed  for  trends  in  Type  of  Research  Perfonned  over  time.  A  description  of 
Mentzer  and  Kahn’s  typology  follows: 

Normative  Literature :  “Research  that  examines  what  ought  to  be  and  what  individuals 
and  organizations  ought  to  do”  (Mentzer  and  Kahn  1995).  “Literature  might  be  cited  in 
the  article,  but  the  point  of  the  inclusion  of  this  literature  is  to  support  the 
opinio  ns/assertions  of  the  author”  (Carter  and  Ellram  2003). 

Literature  Reviews :  “A  review  and  synthesis  of  existing  literature,  the  result  of  which  is 
the  development  of  a  framework,  propositions,  or  normative  prescriptions  grounded  in 
the  existing  literature”  (Carter  and  Ellram  2003). 

Exploratory  Studies :  “Research  that  makes  observations  for  the  purposes  of  developing 
theories,  but  leaves  the  testing  of  the  theories  for  other  studies”  (Mentzer  and  Khan 
1995). 

Methodology  Reviews :  “A  review  of  research  methodologies— a  ‘how-to’  article.  This 
type  of  research  includes  articles  that  review/introduce  an  academic  research 
methodology  as  well  as  a  practitioner  methodology”(Carter  and  Ellram  2003). 

Hypothesis  Testing:  “Articles  that  introduce  and  then  test  research  hypotheses  or 
propositions”  (Carter  and  Ellram  2003). 
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Research  Design  Employed 

The  articles  were  coded  according  to  type  of  design  employed  and  charted  on  a 
graph  to  reveal  trends.  Articles  were  classified  into  one  of  eight  research  designs  that 
were  adapted  from  Carter  and  Ellram  (2003):  Archival  studies,  Interviews,  Topic 
Presentations,  Case  Study(ies),  Experiments,  Focus  groups,  Mathematical  Modeling,  and 
Surveys.  These  categories  are  defined  in  Table  3. 


Table  3.  Definitions  of  Research  Design  Categories 


Design 

Definition 

Topic  Presentation 

Articles  contain  no  discemable  design 
methodology.  Authors  present  subject  matter 
without  explaining  methodology 

Archival 

Data  gathered  from  collective  works 

Case  Study 

In-depth  data  are  gathered  pertaining  to  a 
progam  or  event 

Interviews 

Data  collected  through  the  use  of  interviews 

Surveys 

Data  collected  throughthe  use  of  surveys 

Experiment 

Data  collected  through  an  experimental  process 

Mathematical  Modeling 

Data  collected  using  mathematic  modeling 

The  coders  reviewed  the  articles  and  determined  which  category  was  most 
appropriate  for  each  article  based  on  the  definitions  presented  in  Table  3.  The  data  was 
compiled  and  tables  were  established  in  order  to  determine  the  relative  frequency  with 
which  various  research  designs  were  used  over  time.  To  gain  a  better  insight  of  trends, 
researchers  divided  the  eleven  years  into  the  same  three  periods  discussed  above.  The 
data  was  then  analyzed  for  trends  in  type  of  design  over  time.  Initially  a  chi-square  test 
was  run  to  determine  whether  changes  were  significant,  however  a  “warning”  message 
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appeared  stating  that  more  than  20%  of  all  data  fields  registered  less  than  live 
occurrences.  Therefore,  since  the  results  of  the  chi-square  analysis  were  dubious,  the 
researchers  did  not  rely  on  these  results. 

Analysis  Employed 

The  articles  were  then  organized  according  to  type  of  data  analysis  applied.  Nine 
types  of  data  analysis  were  used  as  categories.  These  categories,  adapted  from  Carter  and 
Ellram  (2003),  include:  No  analysis,  Anecdotal,  ANOVA,  Comparative  Analysis, 

Content  analysis,  Correlation  Analysis,  Descriptive  Statistics,  Factor  Analysis,  and 
Regression.  These  categories  are  defined  in  Table  4.  When  researchers  looked  at  the 
types  of  analyses  perfonned  over  the  eleven  years,  there  was  little  representation  (10 
articles)  from  four  categories.  Researchers  grouped  these  four  categories  with 
Descriptive  Statistics  under  a  new  category  called  “Statistical  Analysis”  as  they  are  all 
various  methods  of  statistical  analysis.  The  five  subcategories  in  Statistical  Analysis  are 
Factor  Analysis  with  5  articles,  ANOVA  with  only  one  article,  Regression  with  also  with 
only  one  article,  Correlation  Analysis  with  3  articles,  and  Descriptive  Statistics  with  2 1 
articles — for  a  total  of  3 1  articles.  Again,  the  data  was  broken  into  three  time  periods  and 
analyzed  based  on  frequency  and  percentage  per  time  period.  Initially  a  chi-square  test 
was  run  in  order  to  detennine  whether  changes  were  significant,  however  a  “warning” 
message  appeared  stating  that  more  than  20%  of  all  data  fields  registered  less  than  five 
occurrences.  Again,  with  the  results  of  this  analysis  in  question,  researchers  chose  not  to 
rely  on  them. 


13 


Table  4.  Definitions  of  Analysis  Categories 


Analysis 

Definition 

No  Analysis 

No  discernable  analysis  conducted  for  the  article 

Anecdotal 

Based  on  incidental  observations  or  reports  rather  than 
on  systematic  evaluation 

Content  Analysis 

A  detailed  systematic  evaluation  of  a  particular  body 
of  material  for  the  purpose  of  identifying  patterns, 
themes,  or  biases 

Comparative 

Utilizing  comparison  as  a  method  of  analysis  (e.g. 
outlining  results  with  a  comparison  between  DoD 
Restructuring  to  that  of  a  civilian  organization) 

Statistical  Analysis 

Uses  statistical  methods  to  analyze  data.  These 
methods  include  ANOVA,  correlation  analysis, 
regression  analysis,  descriptive  statistics,  and  factor 
analysis 

Institutional  and  Individual  Contributions 

The  ARJ  is  the  primary  intellectual  forum  for  the  Defense  Acquisitions 
community.  The  primary  contributors  to  this  forum  are  the  authors  and  the  institutions 
they  represent.  Researchers  looked  at  three  aspects  of  contribution:  institution,  author, 
and  number  of  authors  per  article.  An  examination  of  institutions  will  detennine  whether 
articles  were  of  an  academic  or  practical  nature  and  if  they  came  from  a  government  or 
civilian  perspective.  Examining  author  production  will  give  some  indication  of  thought 
leaders  within  the  community.  Researchers  also  looked  at  the  number  of  authors  per 
article  to  determine  if  there  was  a  trend.  This  information  will  provide  some  indication 
about  the  type  of  article  being  produced  in  the  ARJ. 

To  discover  trends  among  institutional  contributions,  researchers  reviewed 
articles’  author  biographies.  On  the  first  trial,  article  institutions  were  collected.  Some 
institutional  information  from  biographies  was  incomplete  or  vague,  so  researchers  made 
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subjective  decisions  on  institutional  contribution  based  on  the  available  information.  For 


example,  an  article  may  state  that  the  author  was  currently  at  one  location,  but  recently 
graduated  from  a  military  institution  such  as  AFIT.  In  this  instance  researchers  surmised 
that  the  article  was  produced  at  AFIT  and  so  the  article  was  coded  as  such.  At  this  time, 
researchers  created  institutional  category  groupings  and  recoded  the  articles  accordingly 
into  these  categories.  The  institutional  categories  are  defined  in  Table  5.  The  data  was 
then  compiled  and  grouped  into  three  time  periods  and  analyzed  based  on  frequency  and 
percentage  per  time  period.  The  data  was  reviewed  to  reveal  trends. 


Table  5.  Definitions  of  Institutional  Categories 


Institutional  Category 

Definitions 

Civilian  Universities 

All  public  civilian  institutions  of  higher  education 

DSMC/DAU 

Articles  published  by  students  or  faculty  at  Defense 

Systems  Management  College  or  Defense  Acquisition 
University.  Provides  practitioner  training,  career 
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management,  and  services  to  the  Acquisition  Technology, 
and  Logistics  community 

Civilian  Research  Firm/  Defense  Contractor 

Any  civilian  research  firms  or  contractors  (e.g.  RAND, 
Boeing,  MITRE) 

Other  USAF 

Non  academic  Air  Force  institutions  (e.g.  C-17  SPO) 

Other  USA 

Non  academic  Army  institutions  (e.g.  US  Army  Test  and 
Evaluation  Command) 

ICAF 

Students  or  faculty  at  Industrial  College  of  the  Armed 
Forces.  ICAF  is  an  executive  education  program  for  select 
upper-middle  managers  in  the  Air  Force  and  Allied  services 

NPS 

Students  or  faculty  at  the  Naval  Post  Graduate  School. 
Military  Post-graduate  educational  institute 

AFIT 

Students  or  faculty  at  the  Air  Force  Institute  of 

Technology.  Military  Post-graduate  educational  institute 

SECDEF 

All  Articles  originating  from  either  the  office  of  the 
Secretary  of  Defense  or  one  of  the  various  Under 

Secretaries  of  Defense 

USAFA 

Students  or  faculty  of  United  States  Air  Force  Academy. 
Military  undergraduate  educational  institution 

Other  USN 

Non  academic  Naval  Institutions  (e.g.  SPAWAR) 

Other  DoD 

Non  academic  Department  of  Defense  Institutions  which 
cannot  be  attributed  to  one  of  the  services  (e.g.  Defense 
Logistics  Agency) 

National  Defense  University 

Students  or  Faculty  of  the  National  Defense  University.  A 
center  for  joint  professional  military  education 

AC  SC 

Students  or  faculty  of  the  Air  Command  Staff  College. 
Professional  military  education  institution  geared  towards 
junior  field  grade  officers 

Air  War  College 

Students  or  faculty  of  Air  War  College.  Professional 
military  education  institution  geared  towards  upper-middle 
managers 

Other  Gov 

Originates  from  non-DoD  organizations 

USMA 

Students  or  faculty  of  United  States  Military  Academy. 
Military  undergraduate  educational  institution 

Naval  War  College 

Students  or  faculty  of  Naval  War  College.  Professional 
military  education  institution  geared  towards  upper-middle 
managers 

In  order  for  researchers  to  determine  individual  contributions,  articles  per  author 
were  counted  and  a  frequency  table  was  created.  Subsequent  analysis  evaluated  any 
trends  in  these  areas.  A  count  of  authors  per  articles  was  then  compiled  and  an  analysis 
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of  variance  (ANOVA)  was  performed  to  determine  whether  the  mean  number  of  authors 


per  article  varied  over  time. 


Overview 

This  chapter  described  the  methodology  employed  in  this  research  project. 
Chapter  three  will  present  the  analysis  results. 


III.  Results  and  Analysis 


Introduction 
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Chapter  Three  reports  the  analysis  of  the  data  in  order  to  answer  the  investigative 
questions.  This  chapter  examines  subject  categories,  the  type  of  research,  the  research 
design,  and  the  analysis  employed.  It  also  discusses  some  trends  in  contributing 
institutions  and  contributing  authors. 

Research  and  Investigative  Questions: 

This  section  reviews  the  analysis  of  each  investigative  question.  The  first 
question  asks  what  subjects  have  been  addressed.  The  second  question  asks  what  the 
methods  are  by  which  investigators  have  studied  the  subjects.  The  final  question  asks 
what  authors  have  contributed  to  the  ARJ,  and  with  what  institutions  are  they  affiliated. 

1.  What  subjects  have  been  addressed  in  the  ARJ?  What  do  these  topics  suggest 

about  acquisition  reform  in  the  DoD? 

An  examination  of  article  subjects  will  determine  what  the  ARJ  has  presented  and 
will  illuminate  the  topics  that  are  important  to  the  Defense  Acquisition  community. 


Subject  Categories 
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Frequencies 


The  graphs  below  show  the  results  of  subject  coding  of  articles  from  the  eleven 
years  of  ARJ.  Figure  1  depicts  a  graphical  representation  of  the  number  of  articles  within 
each  category,  in  ascending  order. 


Subject  Categories 


Risk  Management 
Buyer/Seller  Relationships 
Performance  and  Measurement 
Organizational  Issues 
Research  and  Development 
Industry  Issues 
Contract  Management 
Policy  and  Regulation 
Program  Fielding  and  Implementation 
Analysis  and  Decision  Making 
Cost  and  Schedule 
Interoperability 
Management  and  Organizational  Behavior 
Acquisition  Strategy 
Reform  Initiatives 


10 


15 


20 


25 


Figure  1.  Frequency  of  Articles  per  Subject  Category 


Table  6  shows  the  number  of  articles  in  a  given  category  and  the  percentage  of  the 
articles  per  category,  per  year. 


Table  6.  Frequency  and  Proportion 
of  Subject  Category  per  Period 


30 
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Reform  Initiatives 
Acquisition  Strategy 
Management  and 
Organizational  Behavior 
Interoperability 
Cost  and  Schedule 

Analysis  and  Decision  Making 
Program  Fielding  and 
Implementation 
Policy  and  Regulation 
Contract  Management 
Industry  Issues 

Research  and  Development 
Organizational  Issues 
Performance  and 
Measurement 

Buyer/Seller  Relationships 
Risk  Management 


Reform  Initiatives 
Acquisition  Strategy 
Management  and 
Organizational  Behavior 
Interoperability 
Cost  and  Schedule 

Analysis  and  Decision  Making 
Program  Fielding  and 
Implementation 
Policy  and  Regulation 
Contract  Management 
Industry  Issues 

Research  and  Development 
Organizational  Issues 
Performance  and 
Measurement 

Buyer/Seller  Relationships 
Risk  Management 


1 

2 

3 

9 

11 

7 

27 

6 

9 

6 

21 

3 

7 

10 

20 

4 

6 

10 

20 

7 

7 

4 

18 

6 

6 

1 

13 

4 

5 

4 

13 

9 

2 

0 

11 

5 

2 

3 

10 

2 

1 

6 

9 

3 

2 

2 

7 

2 

3 

1 

6 

2 

2 

2 

6 

2 

2 

2 

6 

1 

0 

5 

6 

65 

65 

63 

1 

2 

3 

13.8% 

16.9% 

11.1% 

9.2% 

13.8% 

9.5% 

4.6% 

10.8% 

15.9% 

6.2% 

9.2% 

15.9% 

10.8% 

10.8% 

6.3% 

9.2% 

9.2% 

1 .6% 

6.2% 

7.7% 

6.3% 

13.8% 

3.1% 

0.0% 

7.7% 

3.1% 

4.8% 

3.1% 

1 .5% 

9.5% 

4.6% 

3.1% 

3.2% 

3.1% 

4.6% 

1 .6% 

3.1% 

3.1% 

3.2% 

3.1% 

3.1% 

3.2% 

1.5% 

0.0% 

7.9% 

The  top  five  article  categories  including  Reform  Initiatives,  Acquisition  Strategy, 
Management  and  Organizational  Behavior,  Interoperability,  and  Cost  and  Schedule 
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comprised  55%  of  all  articles  over  the  eleven  years  of  publication.  As  expected  for  a 
journal  covering  acquisition  refonn,  Reform  Initiatives  was  the  most  popular  subject  with 
27  occurrences,  about  14%  of  all  articles  published  in  the  eleven  years  of  ARJ. 
Acquisition  Strategy  has  2 1  total  occurrences  including  six  articles  in  the  first  period, 
nine  in  the  second  period,  and  six  in  the  third  period.  To  discover  what  the  peak  in  the 
second  period  might  mean,  researchers  reviewed  articles  in  this  time  period  and  found 
that  all  articles  on  Acquisition  Strategy  in  Period  Three  were  on  various  topics  and  were 
focused  on  different  issues.  There  was  no  discemable  connection  between  the  articles  to 
explain  the  relatively  higher  concentration  in  this  time  period,  so  researchers  explained 
the  peak  as  coincidence.  The  two  categories  Management  and  Organizational  Behavior 
and  Interoperability  were  tied  for  third  with  20  articles  on  their  subject  over  the  eleven 
years.  Management  and  Organizational  Behavior  saw  a  rise  in  time  periods  two  and 
three.  Although  the  number  of  articles  on  this  subject  increased  in  these  time  periods, 
researchers  found  no  systematic  explanation  for  the  spike.  Articles  on  Interoperability 
totaled  20.  This  category  has  been  steadily  rising  from  four  articles  in  the  first  period  to 
six  articles  in  the  second  period,  then  up  to  ten  articles  by  the  third  period.  Researchers 
attribute  this  rise  to  a  greater  reliance  on  computers  and  technology.  Finally  Cost  and 
Schedule,  which  comprised  of  approximately  9%  of  all  articles  in  the  ARJ,  drops  from 
seven  articles  in  the  first  and  second  periods  to  only  four  articles  in  the  last  period. 

The  drop  in  articles  in  the  Policy  and  Regulation  subject  category  is  an  interesting 
trend.  In  the  first  period  Policy  and  Regulation  is  at  13.8%  of  all  articles.  By  Period 
Two  this  category  dramatically  drops  to  just  of  over  3%  and  becomes  nonexistent  by 
Period  Three. 
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The  only  other  dramatic  spike  that  appears  in  Table  6  was  found  in  the  Risk 


Management  category.  The  spike  in  the  number  of  articles  published  appeared  in  the 
third  period  with  five  occurrences.  Only  one  other  article  for  Risk  Management  had  been 
published  prior  to  Period  Three.  This  increase  in  interest  is  accounted  for  by  Special 
Edition  34,  published  in  Summer  2003,  which  specifically  addressed  the  issue  of  Risk 
Management. 

2.  The  next  question  addresses  the  methods  by  which  investigators  have  studied 
the  subjects.  Specifically,  it  asks  how  these  subjects  have  been  explored. 

a.  What  research  methodologies  have  been  used  to  explore  the  subjects, 
and  to  collect  data? 

b.  What  analytical  techniques  have  been  used  on  the  data? 

Researchers  looked  at  three  aspects  of  article  methodology  including:  Type  of 
Research,  Research  Design  (data  collection),  and  Data  Analysis  Employed. 


Type  of  Research  Performed 

Type  of  research  was  analyzed  using  the  categories  defined  in  Chapter  2.  Table  7 
depicts  the  frequency  and  proportion  of  articles  per  Type  of  Research  category,  per  time 
period. 

Table  7.  Frequency  and  Proportion  of  Type  of  Research 
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Type 

1 

2 

3 

Normative 

29 

30 

21 

80 

Methodology 

21 

22 

24 

67 

Exploratory 

1 1 

10 

12 

33 

Hypothesis 

2 

2 

4 

8 

Literature 

2 

1 

2 

5 

65 

65 

63 

Type 

1 

2 

3 

Normative 

44.6% 

46.2% 

33.3% 

Methodology 

32.3% 

33.8% 

38.1% 

Exploratory 

16.9% 

15.4% 

19.0% 

Hypothesis 

3.1% 

3.1% 

6.3% 

Literature 

3.1% 

1 .5% 

3.2% 

Normative  research  made  up  approximately  41%  of  articles.  Methodology 
Reviews  accounted  for  approximately  35%  of  the  articles  and  Exploratory  Studies  for 
17%  the  articles.  Hypothesis  Testing  only  made  up  about  4%  and  Literature  Reviews, 
with  the  fewest  occurrences,  accounted  for  just  2.5%  of  the  articles.  The  only  significant 
trend  in  this  data  was  that  by  Period  Three  Normative  Literature  dropped  from  roughly 
45%  in  the  first  two  periods  down  to  about  33%  by  the  third  period. 


Research  Design  Employed 

Articles  were  analyzed  to  detennine  the  most  commonly  used  research  designs, 
according  to  the  framework  described  in  Chapter  Two.  Table  8  depicts  the  frequency 
and  proportion  of  articles  per  Research  Design,  per  time  period. 
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Table  8.  Frequency  and  Proportion  of  Research  Design 
Design 


1 

2 

3 

Topic  Presentation 

32 

31 

25 

88 

Archival 

19 

14 

14 

47 

Case  Study 

9 

9 

14 

32 

Interviews 

4 

2 

4 

10 

Surveys 

1 

7 

3 

11 

Experiment 

0 

2 

2 

4 

Mathematical  Modeling 

0 

0 

1 

1 

65 

65 

63 

Design 

1 

2 

3 

Topic  Presentation 

49.2% 

47.7% 

39.7% 

Archival 

29.2% 

21.5% 

22.2% 

Case  Study 

13.8% 

13.8% 

22.2% 

Interviews 

6.2% 

3.1% 

6.3% 

Surveys 

1.5% 

10.8% 

4.8% 

Experiment 

0.0% 

3.1% 

3.2% 

Mathematical  Modeling 

0.0% 

0.0% 

1 .6% 

Topic  Presentation  makes  up  approximately  45%,  Archival  design,  24%;  Case 
studies,  16%;  Surveys,  6%;  Interviews,  5%,  and  Experiments,  2%.  There  was  also  one 
article  with  Mathematical  Modeling— making  up  only  0.5%  of  all  articles.  In  Table  8, 
Design  is  depicted  in  three  time  periods  comprised  of  equal  numbers  of  articles:  the  first 
four  years  in  Time  period  1,  the  following  three  years  in  Time  period  2,  and  the  last  four 
years  in  Time  period  3.  Here,  Topic  Presentation  and  Archival  Designs  were  very 
common  in  the  earliest  time  period  making  up  78.4%  of  all  articles  published.  By  the 
third  time  period,  Topic  Presentation  was  still  the  most  common  type  of  design,  but  Case 
Studies  had  become  as  common  as  Archival  Design— for  the  second  most  common  design 
type. 
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Analysis  Employed 

There  were  initially  nine  types  of  analysis  employed,  however,  when  researchers 
looked  at  the  types  of  analyses  performed  over  the  eleven  years,  there  was  little 
representation  (ten  articles)  from  four  categories.  Researchers  grouped  these  four 
categories  with  Descriptive  Statistics  under  a  new  category  called  “Statistical  Analysis” 
as  they  are  all  various  methods  of  statistical  analysis.  The  five  subcategories  in 
Statistical  Analysis  are  Factor  Analysis  with  5  articles,  ANOVA  with  only  one  article, 
Regression  with  also  with  only  one  article,  Correlation  Analysis  with  3  articles,  and 
Descriptive  Statistics  with  2 1  articles— for  a  total  of  3 1  articles.  Once  this  adjustment  was 
made  to  the  categories,  researchers  created  Table  9  to  review  the  data  and  identify  trends 
over  time.  Table  9  depicts  the  frequency  and  proportion  of  articles  per  Analysis 
Employed,  per  time  period. 


Table  9.  Frequency  and  Proportion  of  Analysis  Employed 
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Analysis 

1 

2 

3 

No  Analysis 

29 

27 

26 

82 

Anecdotal 

17 

13 

14 

44 

Content  Analysis 

10 

5 

7 

22 

Comparative 

5 

6 

3 

14 

Statistical  Analysis 

4 

14 

13 

31 

65 

65 

63 

Analysis 

1 

2 

3 

No  Analysis 

44.6% 

41.5% 

41.3% 

Anecdotal 

26.2% 

20.0% 

22.2% 

Content  Analysis 

15.4% 

7.7% 

11.1% 

Comparative 

7.7% 

9.2% 

4.8% 

Statistical  Analysis 

6.2% 

21.5% 

20.6% 

1 

1 

1 

In  Table  9  researchers  saw  that  No  Analysis  made  up  a  large  percentage  of 
articles  per  year,  however  in  year  11,  No  Analysis  is  tied  for  lowest  occurrences  with 
Comparative  Analysis  at  only  7.7%.  Researchers  define  the  “No  Analysis”  category  to 
mean  that  the  article  contained  no  discernable  analysis.  It  is  also  interesting  to  see  that  in 
the  last  two  time  periods  of  the  ARJ,  Statistical  Analysis  is  steadily  making  up  larger 
percentages  of  articles  published. 


3.  What  authors  have  contributed  articles  to  ARJ,  and  with  what  institutions  are 
they  affiliated? 


An  examination  of  institutions  will  determine  whether  articles  were  of  an 
academic  or  practical  nature  and  if  they  came  from  a  government  or  civilian  perspective. 
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Examining  author  production  will  give  some  indication  of  thought  leaders  within  the 


community. 


Contributing  Authors 

Researchers  plotted  the  author’s  data  onto  the  charts  below  to  reveal  trends. 
Researchers  included  infonnation  on  how  many  articles  were  contributed  by  each  author 
and  how  many  authors  there  were  per  article  over  the  eleven  years  of  publication.  Table 
10  presents  the  top  twelve  contributing  authors  from  Years  1  through  1 1 . 


Table  1C 

1.  Top  Contributors 

AUTHORS 

CONTRIBUTIONS 

Christensen,  David  Ph.D. 

6* 

Nissen  Dr.  Mark  E. 

5 

Washington,  William  N. 

5 

Alford,  Lt  Col  Lionel  D.  Jr. 

4 

Arora  Ashish 

3 

Besselman,  Maj  Joseph 
USAF 

3 

Driessnack,  Lt  Col  John 

D. 

3 

Larkey  Patrick 

3 

Linster  Bruce  G. 

3 

Pollock,  Neal 

3 

Snider  Dr.  Keith  F. 

3 

Templin,  Carl  Ph.D. 

3 

*Note:  This  includes  an  issue  introduction 
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Table  1 1  shows  the  proportion  of  authors  per  article  over  the  1 1  years  of  publication. 


Table  11.  Authors  per  Article 

One  Two  Three  Four 


Year  Mean  Author  Authors  Authors  Authors 


1 

1.16667 

83.3% 

16.7% 

0.0% 

0.0% 

2 

1.25 

72.7% 

18.2% 

9.1% 

0.0% 

3 

1.18182 

81.8% 

18.2% 

0.0% 

0.0% 

4 

1.76 

60.0% 

8.0% 

28.0% 

4.0% 

5 

1.66667 

57.1% 

23.8% 

14.3% 

4.8% 

6 

1.65217 

56.5% 

26.1% 

13.0% 

4.3% 

7 

1.68182 

68.2% 

9.1% 

9.1% 

13.6% 

8 

1.28571 

78.6% 

14.3% 

7.1% 

0.0% 

9 

2.21053 

31 .6% 

26.3% 

31.6% 

10.5% 

10 

1.76471 

52.9% 

17.6% 

29.4% 

0.0% 

11 

2.15385 

23.1% 

38.5% 

38.5% 

0.0% 

Researchers  looked  at  the  number  of  contributing  authors  per  article.  This  data 
revealed  that  the  number  of  authors  per  article  appeared  to  increase  significantly  since  the 
beginning  of  publication  in  1994.  In  the  first  year,  a  large  majority  of  articles  were 
written  by  one  author  with  only  16.7%  of  articles  written  by  two  authors.  In  Year  4, 
researchers  saw  that  there  was  a  significant  increase  in  multiple  authors  with  32%  of 
articles  having  three  or  more  authors.  Year  9  saw  a  first,  where  multiple  authored  articles 
out-numbered  single  authored  articles.  By  Year  11,  only  23. 1%  of  all  articles  were 
written  by  one  contributing  author.  In  order  to  run  an  ANOVA,  data  was  grouped  into 
the  three  time  periods  as  established  above.  The  results  suggest  a  statistically  significant 
overall  difference  across  time  periods  (P  >  F  0.0205).  Carter  and  Ellram  found  a  similar 
trend  in  their  study  of  the  Journal  of  Supply  Chain  Management  where  the  average 
number  of  authors  per  article  rose  over  time  (2003). 
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Contributing  Institutions 

Researchers  looked  at  contributing  institutions  as  well  as  contributing  authors. 
Table  12  shows  the  proportion  of  articles  per  contributing  institutions,  per  time  period. 


Table  12.  Proportion  of  Contributing  Institutions 


1 

2 

3 

Civilian  Universities 

14.1% 

24.0% 

28.6% 

DSMC/DAU 

19.7% 

12.0% 

10.4% 

Civilian  Research  Firm/ 

Defense  Contractor 

9.9% 

12.0% 

19.5% 

Other  USAF 

1 .4% 

16.0% 

10.4% 

Other  USA 

12.7% 

6.7% 

7.8% 

ICAF 

12.7% 

2.7% 

2.6% 

NPS 

4.2% 

8.0% 

3.9% 

AFIT 

9.9% 

2.7% 

2.6% 

SECDEF 

5.6% 

5.3% 

1 .3% 

USAFA 

4.2% 

1.3% 

3.9% 

Other  USN 

1 .4% 

5.3% 

2.6% 

Other  DoD 

2.8% 

0.0% 

1 .3% 

National  Defense  University 

1 .4% 

0.0% 

1 .3% 

ACSC 

0.0% 

1.3% 

1 .3% 

Air  War  College 

0.0% 

1.3% 

0.0% 

Other  Gov 

0.0% 

1.3% 

0.0% 

USMA 

0.0% 

0.0% 

1 .3% 

Naval  War  College 

0.0% 

0.0% 

1 .3% 

1  1  1 


As  previously  seen  with  Research  Design  and  Analysis  Employed,  the  analysis  of 
institutions  was  also  aggregated  into  three  time  periods.  Table  12  shows  the  percentages 
of  institutional  contributions  over  the  three  time  periods.  Here,  researchers  could  see  that 
contributions  by  civilian  universities  were  on  the  rise  from  14.1%  in  Period  1  to  28.6% 
by  Period  3.  Contributions  by  civilian  research  firms  and  defense  contractors  were  also 
on  the  rise  from  almost  10%  in  Period  1  to  19.5%  by  Period  3.  While  the  civilian 
institutions  are  rising,  some  government  institutions’  contributions  appear  to  be 
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declining.  For  example,  DSMC/DAU  fell  from  almost  20%  in  Period  1  to  10.4%  in 
Period  3.  Contributions  from  AFIT  fell  dramatically  from  10%  of  contributions  in  Period 
1  to  not  even  3%  in  Periods  2  and  3.  Other  interesting  rises  and  falls  are  seen  in  Army, 
Navy,  and  Air  Force  contributions.  Air  Force  contributions  jumped  from  1.4%  of 
institutional  contributions  in  Period  1  to  16%  in  Period  2,  followed  by  a  decline  in  Period 
3  to  10.4%.  Another  jump,  followed  by  a  fall,  is  seen  in  Navy  contributions  with  1.4%  to 
5.3%,  then  falling  in  Period  3  to  2.6%  of  institutional  contributions.  Anny  institutions, 
however,  showed  a  different  trend;  n  Period  1  they  contributed  almost  at  13%  of  articles, 
then  in  the  last  to  periods  their  contributions  were  in  the  7-8%  range. 

Table  13  depicts  the  percentage  of  articles  published  by  civilian  organizations 
against  government  organizations.  It  also  shows  practitioner  organizations  against 
academic  organizations. 


Table  13.  Proportion  of  Civilian  vs.  Government  Organizations 


Contributor  Type 

1 

2 

3 

Civilian  Organizations 

23.9% 

36.0% 

48.1% 

Military  Organizations 

76.1% 

64.0% 

51.9% 

Table  14.  Proportion  of  Practitioners  vs.  Academics 


Contributor  Type 

1 

2 

3 

Practitioners 

66.2% 

53.3% 

55.8% 

Academics 

33.8% 

46.7% 

44.2% 

These  percentages  reveal  that  government  organizations  consistently  make  up  the 
larger  percent  of  contributions  in  all  three  time  periods.  There  was,  however,  an  upward 
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trend  in  the  contributions  from  civilian  institutions.  By  Period  3  civilian  contributions 
were  at  48. 1%  and  government  contributions  were  at  5 1 .9%. 

There  was  a  similar  trend  in  the  contributions  from  practicing  institutions  and 
academic  institutions.  The  first  period  shows  that  academic  institutions  only  accounted 
for  about  a  third  of  the  contributions,  however  by  Period  3  they  accounted  for  44.2%  of 
all  contributions. 

Overview 

This  chapter  reported  the  analysis  of  the  data  used  to  answer  the  investigative 
questions.  It  examined  subject  category,  the  type  of  research,  the  research  design,  and 
the  analysis  employed.  It  also  discussed  some  trends  in  contributing  institutions  and 
contributing  authors.  Chapter  Four  will  present  the  conclusions  of  the  study 


IV.  Conclusions 
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Introduction 


This  chapter  will  present  the  conclusions  of  the  study.  It  will  address  the  research 
problem  by  discussing  the  results  in  terms  of  the  investigative  questions.  Researchers 
begin  by  restating  the  research  problem  and  then  present  the  findings  with  regards  to  the 
individual  investigative  questions. 

From  the  beginning,  the  ARJ  has  established  itself  as  a  journal  that  exchanges 
ideas  within  the  defense  acquisition  community  and  it  has  become  a  place  to  document 
the  Acquisition  Refonn  Movement  in  DoD.  For  this  study,  researchers  set  out  to  trace 
the  history  of  the  Acquisition  Reform  Movement.  To  first  examine  the  health  of  the 
reform  movement,  the  ARJ  needed  to  be  assessed  as  a  whole. 

Discussions 

The  first  investigative  question  researchers  looked  at  was  subject  category.  In  a 
systematic  effort  to  understand  the  topics  of  interest  in  the  ARJ,  researchers  reviewed  the 
articles  in  the  first  eleven  years  and  charted  the  occurrences  of  each  specific  subject. 

The  ARJ  had  several  interesting  trends  in  subjects  addressed.  This  study 
particularly  looked  a  the  top  five  article  categories  including  Reform  Initiatives, 
Acquisition  Strategy,  Management  and  Organizational  Behavior,  Interoperability,  and 
Cost  and  Schedule.  These  five  subject  categories  account  for  more  than  half  of  the 
articles  in  the  ARJ.  Because  the  journal’s  title,  researchers  expected  to  find  many  of  the 
articles  in  the  Acquisition  Strategy  and  Reform  Initiatives  categories.  Researchers  were 
not  surprised  when  these  two  categories  rated  at  the  top,  making  up  about  25%  of  all 
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articles.  Another  category  in  the  top  five  was  Management  and  Organizational  Behavior 
which  saw  a  rise  in  every  time  period.  The  subject  category  for  Interoperability  is  also 
on  the  rise.  Researchers  attribute  this  to  a  greater  reliance  on  computers  and  technology. 
Cost  and  Schedule,  which  comprised  of  approximately  9%  of  all  articles  in  the  ARJ,  has 
seen  a  slight  decline  in  number  of  articles  in  this  subject  category. 

Some  of  the  other  noteworthy  subject  article  trends  are  Policy  and  Regulation  and 
Risk  Management.  In  the  Policy  and  Regulation  category,  there  was  a  dramatic 
reduction  of  articles  in  the  ARJ  from  Periods  One  to  Three.  The  Risk  Management 
category,  on  the  other  hand,  saw  a  dramatic  spike  in  Period  Three  (Year  10).  This  is 
accounted  for  by  Special  Edition  34,  published  in  Summer  2003. 

From  the  researchers’  perspective  it  is  impossible  to  determine  what  was  driving 
article  selection  for  the  ARJ.  Obviously,  the  type  of  article  submitted  for  consideration 
for  publication  was  driven  primarily  by  potential  authors  aware  of  the  journal.  In  turn, 
the  editorial  staff  acted  as  a  gate-keeper  and  determined  which  articles  actually  made  it  to 
publication.  In  some  instances,  editors  took  a  more  proactive  stance  by  initiating  special 
topics  for  certain  issues,  such  as  Risk  Management  (mentioned  previously  in  Chapter 
Three).  This  was  an  effective  way  to  guide  the  subject  matter  addressed  in  the  journal.  It 
would  seem  feasible  that  the  editorial  staff  could  generate  enough  interest  in  a  particular 
topic  to  fill  an  entire  issue.  In  doing  so,  the  journal  could  continue  to  address  the  issues 
that  naturally  flow  from  the  acquisition  community.  In  addition  to  this,  editors  could 
identify  ‘hot-topics’  and  guide  the  direction  of  journal  content. 

The  second  investigative  question  sought  to  explore  the  methods  with  which  ARJ 
authors  have  approached  the  subject  areas.  This  was  studied  by  an  examination  of  the 
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type  of  research  used  in  articles,  how  authors  were  gathering  data,  and  how  that  data  was 
being  analyzed. 

Researchers  made  the  following  observations  about  the  three  characteristics  they 
explored  with  regards  to  methodology: 

•  More  than  three  quarters  of  all  articles  had  either  Nonnative  Literature  or 
Methodology  Review  as  their  type  of  research.  The  remaining  25%  were 
either  Exploratory,  Hypothesis  Testing,  of  Literature  Review. 

•  In  addition  to  this,  nearly  half  of  all  articles  had  Topic  Presentation  as  their 
research  design.  Fewer  than  15%  of  all  articles  had  a  design  that  could  be 
considered  research. 

•  Topic  Presentation  as  a  research  design  saw  fewer  and  fewer  articles. 
Researchers  interpret  this  as  an  indication  that  from  one  period  to  the  next, 
articles  in  the  ARJ  were  becoming  more  structured. 

When  researchers  considered  contributing  institutions  they  noted  a  tendency  in  the 
ARJ  to  publish  more  articles  from  practitioners  rather  than  academics.  Coupled  with  the 
predominance  of  articles  not  grounded  in  research,  it  seems  clear  that  the  intended 
audience  is  the  practitioner.  This  is  consistent  with  the  Editorial  Mission  as  discussed  in 
Chapter  One.  There  could  be  an  argument  that  the  journal  would  benefit  from  a  stronger 
focus  on  academic  research.  The  problem  with  this  is  that  ARJ  could  run  the  risk  of 
climbing  too  high  in  the  ‘ivory  tower’  and  in  doing  so,  forget  its  audience.  Still, 
researchers  feel  that  there  is  an  appropriate  balance  between  academics  and  practitioners, 
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as  well  as  an  appropriate  amount  of  research  articles.  In  reference  to  trends  noted  in 
Chapter  Three,  it  is  apparent  that  the  journal  is  working  towards  this  balance. 

The  third  investigative  question  sought  to  explore  the  origins  of  articles  published 
in  the  ARJ.  An  examination  of  contributing  authors  and  institutions  showed  some 
interesting  trends,  and  two  areas  that  appeared  especially  significant.  The  first  was  the 
ratio  of  practitioner  to  academic  contributors,  and  the  second  was  the  ratio  of  civilian  to 
government  contributors. 

In  Period  One,  researchers  noted  that  academics,  that  is,  members  of  an  academic 
institution,  made  up  only  about  one  third  of  all  contributors.  By  Periods  Two  and  Three 
this  numbers  was  up  around  45%.  While  the  journal’s  contributors  were  predominantly 
from  practitioner  institutions — those  whose  members  are  engaged  in  applying  knowledge 
rather  than  creating  it — in  the  first  period,  by  the  third  period  academics  made  up  nearly 
half  of  the  institution  contributions.  Researchers  see  this  as  a  good  sign  that  academics 
are  taking  a  larger  role  in  the  ARJ. 

In  the  first  period,  civilian  institutions  made  up  only  about  one  quarter  of  all 
contributions.  In  the  second  time  period  they  made  up  over  a  third  of  contributions,  and 
rising  to  just  shy  of  half  of  the  contributions  by  the  last  time  period.  Again,  researchers 
see  this  as  a  positive  trend,  in  which  civilian  institutions  are  becoming  a  bigger  part  of  the 
ARJ  community. 

These  trends  allow  for  the  ARJ  to  develop  a  broader  perspective  and  a  deep  vein 
of  knowledge  from  a  variety  of  different  areas. 

Limitations  of  Research 
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The  two  main  limitations  of  this  research  were  the  investigator  and  the  method. 
As  the  methodology  was  subjective  in  nature,  there  were  a  number  of  opportunities  for 
the  mistaken  interpretation  of  the  articles. 

Future  Research 

There  are  a  few  recommendations  for  future  research  stemming  from  this  study. 
Future  researchers  should  consider  the  following  recommendations: 

•  A  study  replicating  this  methodology  utilizing  multiple  coders 

•  A  more  detailed  study  of  the  articles  contained  in  the  ARJ. 

•  A  study  to  determine  causality  of  the  trends  identified. 

Overview 

This  study  examined  the  evolution  of  the  Acquisition  Review  Journal  through  its 
first  eleven  years  in  publication.  Researchers  assessed  the  Defense  acquisition 
community  through  a  review  of  ARJ  articles.  It  considered  what  areas  academics  and 
practitioners  have  explored  and  how  they  have  done  so.  This  review  documented  such 
characteristics  as  areas  of  study,  methods  of  study,  and  contributors.  Trends  were 
identified  and  conclusions  drawn  as  to  the  contribution  of  ARJ  to  the  Defense 
Acquisition  community  of  practice. 
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